Efficient Layering for High Speed Communication: Fast Messages 2.x

We describe our experience designing, implementing, and evaluating two generations of our high performance communication library, Fast Messages (FM) for Myrinet. In FM 1.x, we designed a simple interface and provided guarantees of reliable and in-order delivery, and flow control. While this was a significant improvement over previous systems, it was not enough. Layering MPI atop FM 1.x showed that only about 20% of the FM 1.x bandwidth could be delivered to higher level communication APIs. Our second generation communication layer, FM 2.0, addresses the identified problems, providing gather-scatter, interlayer scheduling, receiver flow control, as well as some convenient API features which simplify programming. FM 2.x can deliver 70-90% to higher level APIs such as MPI. This is especially impressive as the absolute bandwidths delivered have increased nearly fourfold to 70 MB/s. We describe general issues encountered in matching two communication layers, and our solutions as embodied in FM 2.x. ∗The research is supported in part by DARPA Order #E313 through the US Air Force Rome Laboratory Contract F3060296-1-0286 and Contract F30602-97-2-0121, NSF grant MIP-9223732, and NASA grant NAG 1-613. Support from Intel Corporation, Compaq/Tandem Computers, Hewlett-Packard and Microsoft is also gratefully acknowledged. Andrew Chien is supported in part by NSF Young InvestigatorAward CCR-94-57809. Mario Lauria is supported by a NATO-CNR Advanced Science Fellowship. †In September 1998, Andrew Chien will be the Science Applications International Corporation Chair Professor at the University of California, San Diego. He can be contacted at [email protected] and 9500 Gilman Road, Dept 0114, La Jolla, CA 92093-0114